Chinese Word Segmentation Accuracy and Its Effects on Information Retrieval

In Chinese information retrieval (IR), word segmentation is an essential prerequisite process to break down the documents into smaller linguistic units or word segments so that they can be indexed for subsequent retrieval. Despite a host of Chinese information systems that are in existence today, little work has been done to study word segmentation accuracy and its effect on IR. This article describes a set of experiments that was conducted in the Division of Information Studies, Nanyang Technological University, Singapore, to explore this issue. Four types of automatic character-based segmentation approaches as well as a manual segmentation were used to index a set of test corpus, thereby resulting in five different indices for use in the IR experiments. The segmentation accuracy of each approach was obtained by comparing the automatic segmentation results with the manual segmentation results. A set of IR experiments provided a measure of IR effectiveness using the traditional measures of data recall and data precision. Statistical analysis was applied to explore the correlation between the segmentation accuracy and IR effectiveness. The analysis suggests that the word segmentation approach do have affect on IR results. In particular, the approach that recognizes the higher number of correct words that include 2 or more characters can produce better precision and recall. On the other hand, the existence of ambiguous words resulting from the word segmentation process adversely affect precision.